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ABSTRACT 


Name of the Student ; Musunur Laxmi Prasad 
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Robot Vision 

Names of the thesis supervisors ; 

1. Dr H Hatwal Mechanical Engineering, 

2. Dr H Karnick Computer Science an^ Engineering. 

Month and Year of submission :May 1991 

This work describes how to grasp an unknown polyhedral 
object with the aid of robot vision. A CCD Camera image is 
processed to form a line diagram. The line diagrams are 
interpreted in the high-vision module to determine and locate the 
grippable surfaces. Grippability is assured if two parallel 
surfaces, with the requisite area, on the object are located. Two 
views <usually taken at 180*^) or more may be required. 

We assume that our objects are convex polyhedra. This 
leads to numerous simplifications with no inherent approximation 
as is usually the case with curved surfaces. Ideally an in^ge of a 
polyhedron would consist of regions of uniform intensities and can 
be conveniently represented by a line diagram. 

We interpret these line diagrams extracted from multiple 
views of the object , with a set of rules based on straight 
forward geometric constraints to locate two surfaces that are 
parallel in space. If found, these planes are examined to see if 
they satisfy all the requirements for a legal grasp configuration. 
Actual gripping requires the determination of both location and 
orientation of these planes in the world coordinate system. 

We do not construct an explicit 5-D model of the object 
and thus we avoid any database matching. Once the location and 
orientation parameters are determined, co-ordinate transformations 
are used to determine the joint parameters for a PUMA- 560 robot 
which was used to grip the object, 
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CHAPTER 1 Introduction 


1 . 1 Introduction s 


Automation and Robotics t 


Mass-production assembly lin^s were first introduced at the 
beginning of the twentieth century (1905) by Ford Motor Company. 
Over the ensuing decades, specialized machines have been developed 
for high volume production of parts. Here the machines and 
processes are often very efficient, but they have 
flexibility. Robots on the other hand offer flexibility to the 
manufacturing processes with which they are integrated. They 
achieve this flexibility because they can be reprogrammed to 
accommodate model variations, product changes or new operations. 


In the beginning robots were used only in hot, smelly and 
hazardous jobs. But today they cover a wide range of 
applications. Some of these are loading/unloading of die casting 
machines, spot welding of automobile bodies, arc welding, 
debarring, sealing, gluing, inspection and in assembly operations. 
One major disacbjantage in using robots is that expensive jigs and 
fixtures are required for positioning the object in the right 
position and orientation so that the robot can pick it up from a 
previously known point. This can be efficiently handled by the 
use of a vision system in the control loop. 

In this thesis we address the problem of how a robot can grip 
an unknown object whose position and orientation are not known a 
priori . 

Robot Vision ; 


Robot Vision studies how the visual sense modality can 
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be used in the control loop of robots to carry out different 
tasks. A vision system analyzes images and produces descriptions 
of what is imaged. These descriptions should capture those aspects 
of the objects being imaged that are useful in carrying out some 
task involving the objects. Thus the robot vision system can be 
considered as an element of a feedback loop that is concerned with 
sensing, while other elements are dedicated to decision making and 
implementing these decisions. 

Robot vision systems supply valuable information that can 
be used in 

Collision avoidancs : Vision is used to detect the presence 
of unknown obstacles to help in the path control for collision 
avo i dance . 

Navigation of rnobils robots : Servo ing errors which can lead 
to deviation from precomputed path for mobile robots can be 
corrected through vision. 

Identification of parts : If a mix of different varieties of 
parts are to be separated, then identification of the individual 
parts can be done using vision. 

Inspection of parts : Vision can be used when components are 
to be checked for flaws or sub assemblies are to be checked for 
missing components etc.<e.g. KEYSIGHT 111 > 

Vision based Manipulation : 

Robot Vision can be used in an effective and economic manner 
to permit Industrial Robots to handle imprecisely positioned or 
unoriented workpieces and subassemblies to compensate for buildup 
of errors in tolerances. This would in effect fine-tune the 
position and orientation of the end-effector in picking up an 
object. Thus the cost of expensive jigs and fixtures, that are 



other wise required to maintain the position and orientation of 
the object can be done away with. Moreover jigs and fixtures are 
only for knoxim. objects. But vision can be used to handle 
altogether new obj&cts that are not stored in the database. 

In the case of separated workpieces lying stably on the bed 
one can get an un-obstructed view of the object. In the worst case 
they may be touching each other. The methods developed for this 
case can be carefully modified to take care of objects on a moving 
bed with variable speed. If the workpieces are occluding each 
other one can introduce a mechanism to scatter the objects to give 
un-occluded views of the objects or use multiple views of the 
scene . 

If the Workpieces are hung on hooks i.e. are being 
transported by some overhead chain conveyor or equivalent the 
methodology developed in the first case may not suffice. This 
definitely requires 3-Dimensional information to determine the 
position and orientation of objects in the World Co-ordinate 
system. 

Consider a totally jumbled container having randomly 
positioned, interlocked workpieces. No clear un-obstructed view of 
any of the workpieces would be available. This could be the most 
difficult situation to handle because, apart form all other tasks 
we have to isolate the image region corresponding to a single 
object. 

If the work pieces are arranged in trays and then stacked in 
a bin, the problem would be to distinguish the workpieces from the 
tray material. Although they may not be positioned precisely in 
the containers the permitted variations in the position would be 
small . 

Robotic Gripping t 

The design and operation of any gripper would be 
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governed by the nature of the workpiece being handled, strength of 
grip etc. There are numerous designs of grippers to cover a large 
range of objects. The simplest one used in industry is the 
pneumatically operated parallel jaw gripper. This can take care of 
most of the pick and place operations. Other grippers include 
3-jaw, magnetic, vacuum etc. And there are some special designs of 
grippers that can handle eggs and bulbs. 

The aim of the present work is to grasp unknown, polyhedral 
objects with the aid of robot vision, using a parallel jaw 
gripper. 

1 « 2 Previous work ( 

We briefly review earlier work under two main headings namely 
Robot Vision and Robot Grasping. 

Complets Vision systems for a specific purpose : 

One of the first projects in which visual information was 
used to plan the motion of an Industrial Robot was COPY-DEMO 121, 
built in 1970 by Patrick Winston , Berthold Horn and Eugene 
Freuder at MIT. It was a simple closed-loop hand eye system that 
uses a mechanical manipulator to build a copy of a structure 
composed of childrens toy blocks. A more sophisticated system 
having practical application CONSIGHT-I 131 was developed at 
GENERAL MOTORS in 1978 by Holland, Rosso 1 and Ward. It was a 
Vision-controlled Robot system for transferring parts from a Belt 
conveyor. No occlusion was considered and the object was lying on 
a belt moving at a variable speed. A one-dimensional CCD array 
was used for the camera and processing was done on binary images 
where only the information of the boundary was made use of. 
KEYSI^rr 111 was another vision system developed by General Motors 
that was used in inspection. Using grey-level pixel data it 
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examined the spring assemblies on the engine heads for the 
presence of valve spring cap keys. 

A vision system 141 that acquires cylindrical workpieces from 
bins was developed at University of Rhode Island (1982) by Kelley, 
Birk and Martins. They considered workpieces partially organized 
and unseparated. 

Ca> Robot vision t 

Robot Vision is a vast area dealing with a variety of topics. 
We present a brief survey of work done in the areas of Edge 
detection. Detection of coioneTS, Determination of orientation, 
shape and location 

CiJ> Edge Detection i 

Edge detection has been one of the most active fields in 
vision. The classic paper by RobertsI1065] showed how to extract 
edges from photographs of polyhedral objects digitized on a 
covered drum plotter. Duda & Hart C19723 considered use of the 
Mo%igh Transform, which is a mapping into the parameter space of 
the line sought after. The results produced by the simple edge 
finding techniques left a lot to be desired. Griffith [19753, for 
example, introduced assumptions about the statistics of image 
features, while Shirai C19753 exploited expectations about the 
scene to focus attention on areas where lines were likely to be 
found. Hueckel [1971,19733 developed a least square fitting scheme 
based on the expansion of the brightness function in a region in 
terms of orthogonal functions. Application of the first derivative 
operator is followed by a search for peak values. A method to 
suppress large values near the local maxima is also required. 
Partly for this reason, second derivative operator became popular, 
since the edge is located where their output is zero. Linear 
first-derivative operators are directional but second-derivative 
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operators can be made rotational ly symmetric. 

Horn C1972] showed that laplacian was the lowest order linear 
operator that preserved all information required to reconstruct an 
image. The Gaussian is popular for smoothing operations for 
several reasons, first it is a rotationally symmetric operator 
that is the product of two 1 -D operators , [Horn 19723. And second 
it has the smallest product of width in the spatial and frequency 
domains. 

Marr C19763 at first proposed a directional operator for edge 
detection. He later claimed that a rotational symmetric operator 
was optimal. The contours where output of the operator passes 
through zero are called Zero Crossings. A new theory of edge 
detection based on finding the zero-crossings of the output of an 
even-derivative operator applied to the image is described by Marr 
and Hildreth [19803. They took the laplacian operator and combined 
it with a Gaussian smoothing filter to produce a rotationally 
symmetric edge operator. Ideally the zero-crossings of the 
filtered output are closed contours. 

Canny [19853 has developed a one-dimensional operator that 
provides an optimal trade-off between localization and detection. 
He designed his operator to minimize the sum of two error 
criteria, namely the probability of failing to mark an edge or 
announcing an edge where there is none, and the distance between 
the detected edge and the true edge. His first error criterion 
also implicitly forces the probability of obtaining more than one 
response to a single edge to be low. Canny restricted his search 
for a solution to the space of linear, shift-invariant operators. 
He confirmed the trade-off between s ignal-to-noise ratio and 
localization and showed that the optimal operator can be written 
as a sum of four complex exponentials. The result contains an 
arbitrary spatial scale factor, so that operators of different 



sizes can be 


des i gned . 

This is still an active area as indicated by. the recent paper 
by Hartley [1985]. 

CiiS> Dot»c tion o/ Corners : 

The earliest corner detection methods involve first 
segmenting the image into regions and representing that object 
boundary as a chain code. Corners were identified where the 
direction changed rapidly 151. Later attempts were directed at 
coming up with a corner detector which operated directly on gray 
level images. These include the one developed by Zuniga and 
Haralick £61, Kitchen and Rosenfeld £71, one by Dreschler and 
Nagel £81 and the one by Rangarajan et al £91. Rangarajan et al 
formulated it as an optimization problem. They modeled the local 
gray level function around a corner point with additive Gaussian 
noise and attempted to find an optimal function representing the 
corner detector which when convolved with the gray level function 
yields a maxima at the corner point. 

Object location, Or-iontation and Shapo ; » 

Image understanding research has produced various 
techniques for extracting information about visible surfaces of a 
scene. Early work was done by many researchers. 

Yuan £101 presented a method for determining the 5-D 
position and orientation of an object relative to a camera based 
on a 2-D image of known feature points located in the object. 
Though this problem is solved in classical photogrammetry making 
use of the collinearity condition. Yuan arrives at a solution by 
exploiting the algebraic structure of the problem, independent of 
the configuration or number of feature points. 

Liu & Faugeras £111 present a new method for 
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determination of camera location from 2-D to ?-D straight line or 
point correspondences i.e. kn©wn feature or control points are 
made use of in this process. 

Horn and Ikeuchi 1121 have presented a system that 
locates and grasps parts from a pile. Using photometric stereo to 
make surface orientation measurements the camera field is 
segmented into isolated regions of a continuous smooth surface. 
One of these regions is selected as the target. of the 
physical object associated with the target region is determined by 
h i stogrammi ng surface orientations over that region and comparing 
them with histograms obtained from prototypical objects. Range 
information not available from photometric stereo is obtained by 
the PRISM binocular system. A collision-free grasp configuration 
is computed and executed using altitude and range data. 

Sandini et al 1131 have designed a system for extraction 
of superficial features from a sequence of range data. It is done 
using depth maps because they rteiy constitute a common 
representation for many different processes such as depth from 
stereo, depth from occluding contours, range finders etc and hence 
facilitate data integration for a complete description of the 
object imaged. Some other works in that direction are Shape from 
Shading CHorn3I14» 151, Surface structure and 5-D motion from Image 
flow Kinematics CWaxman, Ullman3I153, Structure from Motion 
CUllman] etc. 

For computation of depth Faugeras and Heber 1171 have 
tried a method that uses a combination of what is known as Active 
Tx-i angulation and Timo of Flight. Active triangulation uses an 
extra source of light to project some pattern onto the objects to 
be measured thereby reducing the complexity of Sioroo matching . 



9 


1*3 Grasping * 

The problem of Grasping has drawn much attention in 
recent years. Hanafusa and Asada £181 presented a 2-D analysis in 
which a potential function based on the shape of the object is 
used to determine stable positions for the placement of a pointed, 
frictionless finger. Okada £19,201 derived equations of motion 
for a sphere and rectangular box manipulated by these fingers with 
hemi -spher i cal tips. Salisbury and Craig £211 develop a more 
general 5-D analysis in which a Jacobian relates forces and 
velocities of the fingers to an equivalent force and velocity of 
the object. 

Kerr and Roth £221 deal with determination of finger 
joint motion to produce a desired motion of the object and 
determination of the workspace of the hand for special cases of 
planar and spatial hands. Randy C Brost £231 investigated 
automatic grasp planning in the presence of uncertainty. The 
algorithm he presented deals with automatic planning of robot 
gripping motion that is insensitive to bounded uncertainties in 
the object location. Uncertainity problem can be removed if vision 
is used in conjunction. 

1.4 Objective of the present work t 

The aim of the present work is to grasp unknown 
polyhedral objects with the aid of robot vision. First, the view 
taken through a CCD Camera is processed (at PC-AT) to form a line 
diagram. The line diagrams are interpreted in the Hi -vision module 
to determine and locate the grippable surfaces. 

We assume that our objects are convex polyhedra. This 
leads to numerous simplifications with no inherent approximation 
as is usually the case with curved surfaces. Ideally an image of a 
polyhedron would consist of regions of uniform intensities and can 
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be conveniently represented by a line drawing. 

We interpret these line drawings in the Hi-vision module 
(workstation) according to a set of rules to look for the 
existence of two surfaces that are parallel in space. Once this is 
done .these planes are examined to see if they satisfy ail the 
requirements for a legal grasp configuration. Actual gripping will 
require the determination of location and orientation of these 
planes in the world coordinate system. This process of searching 
for parallel grippable planes needs more than one view. The first 
attempt is made through two views taken at 180^ apart about the 
vertical line. 

We do not construct an explicit 5-D model of the object. 
We try to analyze the given object for grippability based on a set 
of rules and thus avoid any database irtatching. 

Once the location and orientation parameters are determined, 
co-ordinate transformations are used to determine the joint 
parameters for a PUMA-560 robot which was used to grip the object. 

1 .5 Organization of' thasls* 

In chapter 2 the Schematic diagram of the whole vision system 
EYE KANPUR is presented with a brief description of the hardware 
detail of machines/equipment involved in the Low-vision module. 
Also details of implementation of the Low-vision module are 
presented .Chapter 5 outlines the Hi-vision module of the system 
where the details of rules implemented together with the Camera 
transf ortrat i on to map the camera coordinates to the World 
Coordinate System are discussed. The validity and applications of 
the results are discussed in Chapter 4. Conclusions and directions 
for future work constitute Chapter 5. Appendices deal with the 
specifications of the equipment (Appendix A for camera and 
Appendix B for frame grabber) . 



Chapter 2 Low-vision Module 


2* 1 Introduction s 

A schematic of the hardware layout has been shown in Fig Z.l. 
We briefly describe the methodology followed and the hardware 
requirements. The vision system EYE KANPUR can be broadly 
classified into Low-vision and Hi-vision modules. The Low-vision 
module essentially does image processing and finally produces a 
line diagram of the object imaged. The line diagram is sufficient 
to represent the image of a convex polyhedron. The Hi-vision 
module analyzes this line diagram to , ascertain the grippability 
and then determines the position and orientation required by the 
robot to grasp the object. The Low-vision module of Fig 2.1 is 
discussed in this chapter and the Hi-vision module is explained in 
next chapter. 

2*2 Hardware coin{>onant.s of Low-vision modulo » 
i> A CCD Camera for the imaging. 

ii) A Frame Grabber to digitize and store these images in a 

frame memory 

iii) IBM PC-AT for housing the Frame Grabber, 
iv> ETHERNET card for the communication between PC-AT and 

Workstation through ETHERNET. 

2.2.1 Comorot 

This consists of PULNiX TM-560 camera with an attached 
lens system. The inre-ger is 8.8<mm) X 6.6(mm>and the CCD array is 
of size 500(H>X 582(V>. 

The resolution in Horizontal direction equal to 

— i..e. 0.0176 mm 

«oo 

The resolution in vertical direction is 
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Low-Vlslon Module t 

1. Capture an Image 

2. Detect Edges 

3. Thinning 

4. Segmentation Line Detection 

5. Formation of LINE DIAGRAM 


HI -Vision Nodule » 

1. Formation of Valid Visual 
planes 

2. Ascertain the grippability 

3. Determine the position 

and the orientation (OAT) 
for PUMA to grip the object 


Work Station 
(68030) 

25 MHz 8 Mb 


E 

T 

H 

E 

R 

N 

E 

T 


Fig 2.1 EYE KANPUR Hardware Configuration 
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— ^ i . R . 0.0115 mm. 

9112 

These values are used in converting the pixel units to mm be 
sed in camera transformation. 

Other specifications are given in Appendix A 

2.2.2 Fr-ami& Grabber : 

The PCVISIONplus Frame Grabber (Imaging Technology , USA) 
s a video digitizer and frame memory capable of digitizing 
standard RS-170/550 or CCIR <TM 560 gives a CCIR output) video 
input and storing the digitized image in a special on-board frame 
nemory. The image can be simultaneously displayed on video 
Tsoni tor . 

The PCVISIONplus is a single board that plugs directly into 
an expansion slot in the IBM PC. It digitizes the incoming video 
signal to eight bit accuracy at a rate of 50 frames per second and 
stores the resulting pixels in frame memory. Each pixel is one of 
256 gray levels. Two 512 X 512 or One 640 X 512 image can be 
stored. Display logic converts the pixels in the frame memory back 
to analog RS-170 format for display on a video monitor. There are 
three output channels for pseudo color display. The one input and 
three output channelseach contains eight look-up tables (LUTs). 
Each of these 52 tables contaiPS256 entries. They are helpful in 
performing transformation of the 256 intensity levels, i.e. simple 
point transformations can be done without any processing or delay. 

Further specifications are listed in Appendix B. 

Software for the PCVISIONplus is supplied as a callable 
C library of image processing routines called ITEX library.^^^ ^ 
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2.2.? IBM PC-AT 

An IBM PC-AT or 100% compatible with following specification 
is needed for housing the frame grabber : 

8-bit data bus, 24-bit address bus that allows mapping into 
extended address space. 

16 I/O mapped control registers mappable to any 16-byte 
boundary . 

2.2.4 PCNFS 

PCNFS is the PC version of Sun Microsystem's Network File 
System. It allows sharing of files between users of different 
operating systems in the NFS network including Unix and VMS. 

Essentially it gives us the flexibility of using either of 
the machines without being bothered by DOSs’ limitations. Thus we 
have a general platform over which one can implement any algorithm 
in robot vision. Particularly a 680?0 based workstation for 
Hi-vision module gives the much desired speed. 

2* 3 SSoft-warA dov»lop«d for Low— vision modulo * 

In this section we present the algorithms that we have used. 
The Low-vision module takes in gray level image of the object and 
gives line diagram as output. The various steps involved are 

Jma^o <xc<f'uici tion 

Edg& d&t»ction 

Thiyvning 

Extraction of Contours 

Rosot Contour 

Extraction of linos 

Lino Joining 



15 


We assume that the object is a convex polyhedron and is the 
nly object in the scene. 

2.3. Imag» CLC<fxiisitior\. : 

Image acquisition is done by the library routine grab to 
;ontinuously grab the image and display it on the monitor. It can 
)e frozen at a chosen instant by snap to store the image in the 
‘rame memory. Now we have the image in 512 X 512 array i.e. the 
•'rame memory. 

2.3.2 Edge detection. : 

We have used sobel* s edge operator, from the ITEX 4ibrary to 
find the edges for the following reasons : 

•It is less sensitive to noise than a second-derivative 
operator . 

.Works better than other gradient operators like Roberts and 
gives finer and continuous edges (due to the size of the kernel 
being 5X3) 

•It can be implemented readily in any hardware and if we can 
get a hardware scroll window of size 5X5 we can implement 5X5 
kernel with ease and obtain satisfactory results. 

The Scaling factor would scale the output of the sobel 
operator. Scaling is done by dividing the pixel value with this 
factor. A more detailed discussion on the effect of scaling factor 
is presented in chapter 4. We observed that a factor between 40 
and 60 was performing well for the present lighting conditions. 

2*3.3 Thinning : 

In thinning all that we are now doing is to replace a 
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lor 1 zontal wal 1 
sl+tl/Z at same 
coordinates are 


of thickness tl starting at si by a 
y. This is illustrated in Fig 2.2 
explained in Chapt 0 T 4 , Fig 4.1). 


point at 
< Image 


Algorithm ; 

For y=0 to 512 

Read a horizontal line (512) pixels into a 
For x=0 to 512 

if linbufCx3 = 0 4 linbuf[x + n>0 
C 


thick=l /* wall thickness */ 
begin=x /* beginning of wail ♦/ 


if linbufCx3>0 4 1 i nbuf Cx+l 3>0 thick++; 
if linbufCx3>0 4 1 inbuf Cx+l 3=0 


1 inbufC5123 


if thick=l 

wp i xe 1 (beg i n , y , 200 ) ; (write into a pixel) 

else 

... .. thick 

point=begin+ — — 


wpixel ( point, y, 200) ; 


3 


Thus this can thin down edges to a single pixel width. A 
strategy that identifies the blob (inhere a cornor of the 
polyhedron exists) and handles it in a different manner could give 
us a better skeleton that is closer to the original image. More 
details on desired improvements in thinning are given as 
suggestions for future work in chapter 5. 


2*3*4 Contour Finder : 

This module is called SEGMENT in the code . SEGMENT uses a 
SE13MENT_THRESM0LD which is the allowable deviation for a pixel to 
lile on a contour, i.e. from the current pixel another pixel within 
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Tia deviation will be picked up to form the next point in the 
ontour. If there is more than one pixel falling with in this 
hreshold then the nearest pixel is chosen. Thus in a top-left to 
ottom-right scan one of the contour is detected. If the input 

i.e. the thinned image ) is of the form shown in Fig 2.5 (a) then 

ig 2.3 <b) shows a possible sequence of contours to be detected 

hen SEGMENT is called repeatedly. The sequence of contours 

etected may be different when an actual digital input is given, 
xperimental details are given in chapter 4. 

Algoi'ithm. : 

for j=0 to 511 (.y-directiorO 
for i=0 to 511 (.x-dirBction) 

get hold of a lighted pixel and then mark it as xl,yl; 

<we are interested in just one) 

if (any other lighted pixel falls within THRESHOLD to <xl,yl>> 
mark new point for reference 
else next j ; 

There are some more implementat ional details that are not 
presented here but can be found in the comments along with the 
code. Globaly is a restart point for this entire process so that it 
would not start all afresh from the top. 

2*3*S Ri»s»t Contour : 

Here pixels found in the contour are switched off. This helps 
in getting the ond condition for processing the picture. When 
there is no other lighted pixel the cycle consisting of 

Dotoct Contour 

Rosot Contour 

Lino Findor terminates. 



a 


a 




First Scan 


Second Scan 


Fig 2^3 CbD Possible output of segmentation module 

After repeated scans 



Third Scan 


Fourth Scan 
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2. 3« 0 Lin» finder ; 

This module is called NEW_SEPARATE in th» code. It uses a 
EW_SEPERATE_THRESHOLD which is the allowable deviation for any 
ixel to lie on a digital straight line. It is illustrated in Fig 

.4 <a) where b can not become a member of the digital straight 

ine from <a to c because it exceeds this threshold. Moreover b has 
he maximum deviation from the line joining a c . Hence b is 
arked as a corner in the contour a b c. In a contour C given out 
y SEGHEhn" } scanning is begun from the top point. The point 
laving maximum deviation from the line joining top and bottom 
loints of the contour ( and deviation greater than 
IEW_SEPERATE_THRESHOLD > is selected to be the corner. This 
)rocess is recursively done for the two parts above and below this 
:orner in the entire contour. 

Thus given a contour as input as shown in Fig 2.4 (b> . the 

algorithm separates the valid line segments as shown in Fig 2.4 

(c) by finding all the corners in the first and then joining them. 

2.3.7 Line Joining : 

Once all the lines in the given image of a polyhedral object 
have been found the line diagram of the object has to be formed. 
This module is named OPTICAL_FLOW in the code. 

When a line diagram of an image of a polyhedral object is to 
be formed one has to see that lines that intersect in space 
emanate from the same corner in the image. This is necessitated 
due to the working of the Contowr_d 0 tector & Line Finder that 
uniquely assign any corner to just one line. So, this module finds 
the intersection points of each set (all those intersecting at 
that corner in space) and all the lines are updated with the 
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since b 
it becomes 


the max deviation 
a corner 


- deviation 

D > NEW_SEPERATE_THRESHOLD 
has 



b> input-contour 


C> output- lines 


Fig 2.4 Line Finder 





corresponding new points. Fig 2.5 (a) shows the output of Line 
finder (i.e. input to the line joining module >. The output of the 
line joining module is shown in Fig 2.5 (b>. 

2.3.8 Labeling : 

When one view is to be correlated to another view, we need to 
know the correspondence between the lines <or points in the 
image>of two views. At present we are solving this correspondence 
problem interactively by labeling the lines. <i.e. the labeling is 
done by the user) That is once a line diagram of an object is 
formed the labels for each line are determined interactively and 
stored along with the line. When another view is taken the labeling 
process should be done consistently i.e. the same line should be 
given the same label. These labels are used by the Hi-Vision module 
to ascertain grippability and also in the final determination of 
the position and orientation of the object to be picked. Fig 2.5 
<b> shows a labeled line diagram of a cube. 



a) input :disjoint lines 


b> output ; Joined lines & 


Fig 2 


labe led 


5 Line Joining and Labeling 



Chapter 3 High-Vision Module 


3.1 Introduction * 

In this chapter we deal with the analysis or 
interpretation of the lin© oLiagrcm. obtained from the low-vision 
module. The implemented algorithm is presented in detail. Since we 
are looking for the existence of grippable surfaces without 
identifying the shape completely, the presented algorithms are 
much simpler than those which require matching against a database 
of models of objects 1121. 

3.2 TERMINOLOGY : 


Although the terms and abbreviations used are either standard 
or explained in the body of the chapter, they are listed here for 
conven i ence . 


2- D 

3- D 

bach track 
badp 

blind alley, l&af 
cdr 


node 


grand parent node 
parent node 
root node 
successor node 


pop 


Titio dimensional 
Three dimensional 
to recede in the search process 
Bad predicate 

a node that does not haue any successors 
in lisp this returns the given list 
except the first element, i.e. cdr Ca 6J>=C6J> 
any point under consideration 

predecessor of a parent node 
immediate predecessor of a node 
starting node of the search 
node that emanates from the given 
node 

for a list 1st element is removed 
stack i.e. removes the top most element 
Temporary storing location for plane 


temp plane 
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tho-t is being formed 
World coordinate system. CcartesiarO 
list of nodes that have been traversed 
sin O 
cos O 
sin A 
cos A 
sin T 
cos T 
sin 6 
cos 6 
sin 4> 
cos 4> 

3. 3 Definitions * 

In chapter 2 the final line diagram formed is represented as 
a DCEL. The information in this DCEL is used as an input for the 
further processing. 

DCEL CDovbly Connected Edge List>: 

It is a list of edges. Each edge of the line diagram is 
represented as a list of it-'s associated label, slope, starting 
and end points. 

Fig 3.1(a) shows a line diagram and it-'s associated DCEL. 

Fig 3.1(b) shows the associated graph of the line diagram, if 
P is arbitrarily chosen as the starting node. All the following 
definitions are explained with reference to this graph. 


WCS 

visi ted nodes 

SO 

CO 

SA 

CA 

ST 

CT 

se 

ce 

S4> 

C4> 
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S%icc€^ssor nod^ ; 

It IS the node that emanates from a given node. in Fig 

5.1(b> at the beginning of the graph P has P and P as its 

2 13 

successors . 

Go-alp CGoal Pr^dicat&y : 

In graph search a condition that prevents further search for 
successors at a given node is called a goal predicate. 

Visi-ual Pl<xn& : 

Any apparent plane, which may be a likely candidate for 
a plane in WCS is known as VISUAL PLANE. Fig 5.Z<b) shows the 
visual planes for the line diagram in Fig 5.2<a>. 

VctlidL Pl<xn& : 

Visual planes that correspond to actual 5-D imens i ona 1 
planes are called VALID VISUAL PLANES. Fig 5.2(c) shows this set 
of planes for the line diagram in Fig 5.2(a>. 

3.4 PROPOSITIONS * 

1. A pian& is said to have formed if circular 

connectivity of a chosen node is achieved. 

2. A visual plane thus formed is valid if circular 
connectivity is achieved through minimum number lines. 

3. Two distinct valid visual planes of an image of 
a polyhedral object cannot have more than one line in common, 
i . e . i ntersect i on of two planes in space is along a unique line. 

3.5 Algorithms t 

The High-vision module has been implemented in Common 
Lisp on a MC68050 based workstation running Unix V.5.2 . A 

PUMA-560 robot arm is used to do the actual gripping. 
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The line drawing is a 2-D projection of 5-D surfaces. We 
isolate these 5-D planes and then examine these planes to see if 
they form grippable surfaces. Once these are determined one has to 
determine the position and orientation of these planes in WCS. 
Then the corresponding joint variables for PUMA are determined so 
that it can REACH OUT and GRIP the object. 

The main steps in the algorithm are as follows ; 

Fox'ma.tiorx of xxxlicl ‘oisxjLO.l pianos 
tiJ> S&arch for grippable sxrr faces 

Determination, of position and orientation of 
GrippaJble svtr faces 
ix>J> Determination of joint x>ario±>les 

3.6 Formation of Valid Visual Planest 

An algorithm is presented for the formation of valid visual 
planes of an image of a CONVEX POLYHEDRAL object. The do it 
yoxxrself methodology employed in the field of Artificial 
Intelligence seemed to offer potential advantages CULLMAN, 19791 . 
This algorithm is a depth first search with goalp defined as a set 
of conditions. The search at any node is abandoned if goalp is 
true. When the entire graph is searched, all the visual planes are 
formed. These visual planes are filtered using propositions 2 and 
? to form all the valid visual planes. The method helps in finding 
the interrelationships of 2-0 planar projections of a 5-D object 
which might be totally unknown to the system. 

The working of this module is explained with the help of Fig 
5.2 (a) <b) and <c>.<but the actual results are given in terms of 
lists involving the labels and nodes) 

If the input fCEL to this module is corresponding to the Fig 


the 
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. 2 (a> then first the visual planes are formed. Some such 
possible visual planes are shown in ^19 5.2 (b). These visual 

planes are then examined to give out the valid visual planes as 
shown in Fig 5.2 (c). 

ASSUMPTIC»1S I 

1. Objects are convex polyhedra. 

2. The scene has only one object. 

5. Every line in the image corresponds to an actual physical 
edge of the object <i.e. there are no shadows ) and has a 
representation as a doubly connected edge. 

4. The edges are labelled absolutely. This implies that an 
edge occurring in two or more distinct views has the same label 
and finding correspondences becomes trivial. 

GOALP CONDITICWfS * 

1. When a successor node equals its grand parent node. 

2. When a successor node equals the root node. 

5. When a successor node equals any of the visited nodes. 

4. When a node is a blind alley.i.e. it does not have any 

successors . 

ALGORITHM t 

Phxxs& 1 ; Formation of all visual planes 

input is DCEL and output is VISUAL PLANES 
begin 

planes < — ml 
choose n (any node> 

function findcycle <n ; node , temp_p lanes : list of lines) 
mark n ; 

S non-parent successor (n> 
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V S € S 
if <marked <s) 

{ 

add line (n,s) to temp_p lanes; 

find cycle in temp_planes and add it to planes 

remove line <n,s) from temp_p lanes; 

} 

else 

add <n,s) to temp_planes; 
find-cycle < s , temp_plane ) ; 
end ; 


Ph.cts& Z rFormation of valid visual plane set (WPS) 

input is VISUAL PLANES (VP) 
output is VALID VISUAL PLANE SET (WPS) 

begi n 

WPS <— NIL 
while VP ^ NIL 
do 
f 

VVP 4 — smallest cycle in VP 
delete all C € VP ^ C n VVP > 1 
put VVP into WPS 
> 

end ; 

3.7 6rippabl& surfaces* 

A Grasp Configxtration 1131 should satisfy the following two 
cond i t ions . 

1. Given the gripper's shape and object's shape, it should 
produce a mechanically stable grasp. Such configuration will be 


I r., KAiNjpyq 
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called l&gal ^rctcp configurations. 

2. Such a configuration must be achievable without collisions 
with other obiects. Grasp configurations are limited by the 
relationship between the shape of the gripper and the shape of the 
neighboring obstacles. Such configurations will be called 
coUtsxon fr&e configurations. 

Legal Grasp Configuration * 

Several definitions have been given by Hanafusa & Asada, 
Brady, Boissonnat etc. In the current work we are restricting our 
selves to finding the L&gal Grasp ConfigivTatiorts as given by 
Lozano-Perez t 34, 25J . 

Legal grasping configuration is the one in which the object 
satisfies the following two conditions. 

1. The object is not translated while the gripper is grasping 
the object 

2. The object is not rotated while the gripper is grasping 
the object. 

Legal grasp configuration can be achieved in following steps, 

1. If the centre of area of the object does not coincide with 

the mid point of the two jaws of the gripper, then the 

simultaneous closure of the jaws would result in translation of 
the object. This is illustrated in Fig 5.5 <a> Ci3. 

If the gripper is aligned in such a way that the centre of 

area of the object coincides with the mid point of jaws then the 

simultaneous closure of the jaws would not result in any 
trans lat ion . Thi s is illustrated in Fig 5.5 (a) CiiJ. 
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2. For preventing the object’s rotation check for the lateral 
off-set between the two planes must be done to ensure that the 
planes have a overlap equivalent to the minimum grip area 
requ i red . 

In Fig 5.3 <b)Ci3 there is no overlap between the planes to 
be gripped and would result in rotation of the object. 

In Fig 3.3 ^b)CiiJ the same ob j ect i s shown but the gr i pp i ng 
is being done along the other set of planes. This would not result 
in rotation of the object since there is an overlap between the 
planes to be gripped. 

These can only be implemented after the coordinates in the 
WCS are known. In our work we are only calculating the center of 
area and orientation so that the gripper may be aligned without 
translating the object. 

To determine a set of grippable surfaces, first one has to 
arrive at two parallel surfaces. 

Two planes are parallel if there exists two lines (not 
mutually parallel) that are parallel between them. 

Broadly the method of establishing the parallelism 
between two planes plan® .plan© is as follows. 

let plane^ = ( lp> consist of p line segments 

and plan® =( m ,m ,m ... m_> consist of q line segments 
let count=0 
for i= 1 to p 
for j = 1 to q 

if there exists 1. such that 1. II m 
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£ The parallelism between two planes is known by checking up the 
plines which is the list of sets of parallel lines. 

These sets are made in the following manner : 

In each view after the valid planes are formed the parallel sets 
are formed with in the planes, by looking at their slopes. When a 
line is common to two planes the relationship gets extended and in 
a similar way across the views } 

count=count+l ; 

remove all lines parallel to 1. from plane, and 

V 1 

remove all lines parallel to m from plane^ 

else next j 

if count=2 then the planes are parallel 
return 

else next i 

return nil (planes are not parallel) 

In the actual implementation this was achieved in 
two stages. 

First, only those set of planes that emanate from two 
parallel lines in another plane are considered. Next checking is 
done to see if there exists a different line that is parallel 
between them. 

Information from a difforont viow of th» obj&ct : 

When the system fails to come through the grippability 
analysis in one view it is essential that the information 
available in the second view be correlated with that of the 
previous one. The data-structure keeps the relationship in terms 
of parallelisms between the lines only, since this relationship 
can be extended to any arbitrary view. But this is also the short 
coming of our approach since we can not assure completeness of the 
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algorithm. Because, if a line is not parallel to any other line in 
one view and if this is not visible in the next view it can not be 
correlated to any other line. i.e. the information about the 
polyhedron is not complete. This can be taken care of by using the 
absolute orientation of the edges in space. Which means depth 
information for all those points would be required, hence would be 
a computation intensive operation. 

3.8 Object Position and Orientation in WCS * 

Once the algorithm determines the grippable surfaces, the 
robot has to reach out and grip the object. Therefore the location 
and orientation of the central plane (between the grippable 
surfaces ) has to be transformed from the camera image plane to 
the WCS. This transformation procedure is explained below. 

CiJ> Homjog&n0o\is coordinates : 

For any world point (X.Y.Z) the homogeneous coordinates are 
defined as <kX , kY , kZ , k> , where k is an arbitrary non-zero 
constant. In the vector form 

■ X 

if W = Y 
Z 

then the homogeneous coordinates can be written as 
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Cii^ P»rsp»ctiv& transformation : 


Fig 5.4 (a) shows a simple lens and P is 
P . The focal length of the lens is If the 
point P are x.y.z and that of the image point 
in the figure, then we get 


the image of a. point 
coordinates of the 

» I ? 

are x ,y , z as shown 


11 1 



This can be written in the matrix equation form as 


where 



X 

y 

z 


1 


represents the homogeneous coordinates 
of the point P, 


{ 



kx’ 

ky’ represents the homogeneous coordinates 
kz' of the image point P’ , 


k 
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and P 



is the Perspective Transformation. 


Thus if the image coordinates of a point are known then the 
coordinates of that point can be obtained from 

{*h} =[’■ ] < j.-i) 



However the depth information z’ from the image is not of 
much use because the camera will have a finite depth of field and 
hence the objects at different distances will be imaged sharply at 
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the same z’. Instead. ,t is assumed that z value is obtained 

through other means (e.g. by ranging sensor or stereo vision) and 

then z' IS calculated from the last equation of (5.2). In other 
words, the simple lens has been modeled as a pm hole camera, with 
the pin hole at the lens centre, as shown in Fig 5.4(b) and we can 
still use ( 5.4) whe re the depth i nf ormat i on about z has to be 
supplied separately. 

CiiiS> Trctrts format ion b&txi>&&n Camara coordinat&a and. xix>rld 
coordinates : 

Let us consider Fig 5.5 which shows a world coordinate system 
( X,Y,Z ) and a camera coordinate system (x,y,z). The camera is 
mounted on a gimbai which allows pan through an angle & and tilt 
through an angle a. The pan angle 9 is defined as the angle 
between x and X axes, and tilt, a is defined as angle between z 
and Z axes. The offset of the center of the gimbai from the origin 
of the world coordinate system is denoted by vector , and the 
offset of the center of the imaging plane with respect to the 
gimbai center is denoted by a vector r. 

Suppose that, initially the camera was in normal position, in 

the sense that the gimbai center G and origin of the image plane 

were at the origin of the world coordinate system, xyz was 
coincident with XYZ. Starting from normal position, the 
geometrical arrangement in Fig 5,5 can be achieved in a number of 
ways. We assume the following sequence of steps. 

(1) Translate the gimbai centre G (and hence xyz) by the 

displacement vector where X^,Y^,Z^ are the components of i n 

WCS. 

(2) Pan the x-axis by rotating current xyz about the vertical 
axis by an angle &. 

(5) Tilt the z-axis by rotating current xyz about the current 
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x-axis by an angle a. 

^ (4> Translate the new xyz axis system by displacement vector 

r where , r^ are the components of r in XYZ system. 

The components of a world point # (Fig 5.5) can either be 
expressed as homogeneous coordinates in XYZ or xyz system. These 
components are related by. 




Moreover, if 




are the homogeneous coordinates of 


image point then combining 5.4 and 5.5 we can write 


the 



. . ( 5.6) 




Pou^ ( ©) 



Fig 5.6 
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The three non-trivial equations obtained by term wise 
matching <of right and left sides of 5.6) and solving for X.Y.Z 
are 


‘■X J 

Knowing Z we can calculate z by the following relation. 


<y + r >Sa+r Coi+Z -Z 

g % €> 

Z-r,So»-Ca(X + r >-Z 
2 a o 


. . < 3.7 b> 


x*C6»-y*CotSe>+^ r^C6>- r,C«sa+S«S0(X+r )+X Ur Ce-r OS^+r SaSe+X 

A|^i 2 3 0 jl 2 a o 


(- + 1 ) 


< 3.8) 




x*S0 + y*CaC0+f<r SB +r CaC0-SaCe(X+r )+Y V+r se+r CaC©-r SaC©+Y^ 
' Xlit 2 8 of 12 3 O 


1+r 5 


(- + 1 > 
X 


( 3.9 


One should have the depth information to calculate X, Y and 
we are supplying it in the following manner. The work table height 
is known a priori. The height of the object is also known. When 
the labeling of the line drawing is done, care is taken to see 
that the lines that form an umbrella to the object are labelled 
with a prefix t top^ . That essentially gives the altitude 
of all the lines on the top plane or umbrella plane of the object. 

Thus in Fig 3.6. we take two lines (e.g. til and tl4> that 
lie on the top of the object (hence Z is known) and are members of 
grippable planes. So. we can calculate the fowr world points 



44 


correspond. ng to these tv.o lines that are helpful in ' determining 
the c&nt&r of ar&a and orientatton of plan&s. 

If til has fXl.Yl) (X2.Y2) as its end points 
and tl4 has (X3.Y5) <X4.Y4> as its end points 

X = { X +X +x +X >/4 

Gwnirm 12 0 4 

Y a ( Y 4Y 4Y 4Y)/4 <310) 

t M 0 4 \ ? • xv / 

Then the angle ^ <Fig 3.7 (a) > can be calculated by 

< 3.11) 

We are considering only vertical planes so that 4> suffices 
for or . entat i on . 


4> = atan 


{ 


D^t^rmindiiort, of Joint anglos for- PUMASGO ; 

PUMA-5'60 needs the specification of the world point in the 
form of <X,Y,Z,0,A,T> . X,Y,Z specify the position of a point 
c&ntr<xl ty located between the finger tips in world coordi nates , and 
OAT specify the orientation of the hand 1261. X.Y.Z of the 
centre point are obtained from (3.10). The orientation of the 
grippable planes is obtained in terms of from (5.11). This angle 
4> is used to determine the O, A,T angles in the following way. 

With OAT the rotation matrix [ R 1 is given by 


COSO-SOSACT 

COCT+SOSAST 

L J 

SOCA 


SOST+COSACT 

SOCT-COSAST 

-COCA 

(5.12) 

-CACT 

CAST 

-SA 
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Fig >.7 (a) depicts the tool frame X^Y Z attached 
gripper. Let us consider Fig 3.7 (b> which shows the 
aligned with the object being gripped. 



-S^ 

0 



(3.13) 


The term wise matching of (3.12) with (3.13) yields 
O-T = <^+90 

Taking T=0*^ and A=QO^ , we get 
O = <p +90 

Finally the inverse kinematics is done by the 
subroutines of VAL to get 6 to B . 


to the 
gr i pper 


i n-bu i It 




Fig>.7(a)Tool 


frame definitions 



Fig 3.7(b) 

o-BaSe Frame 
t-TooI Frame 


object 


Chapter 4 


RESULTS AND DISCUSSION 


Introduction i 

In this chapter we present the results of implemented 
algorithms of chapter 2 and ?. First a procedure is discussed to 
calibrate the camera frame. Then the three objects namely a cube, 
a hexagonal prism and a triangular prism, are used to illustrate 
the sequence of operation to determine the gr ippabi 1 i ty . If the 
objects are found to be grippable then RJMA-560 is used to grip 
the objects. 

Calibration t 

The frame grabber or monitor coordinates are shown in Fig 
4.1. The camera coordinate system is chosen such that a movement 
along negative x-axis of camera is shown along positive x-axis on 
the monitor. Similarly the y-axis of the camera was established 
This takes care of the inversion being done by the hardware from 
the image plane (i.e. the CCD array) to the frame grabber <i.e. as 
seen in the display monitor). 

In chapter 5, we obtained explicit equations < 3.7 to 5.9) 

for determining world coordinates <X,Y> of a image coordinate 
<x,y) with known Z. The implementation of these equations require 
knowledge of focal length <X), camera offsets <r and i^^>, angle of 
pan W) and tiitCa). In the present work we have kept camera fixed 
at same location and orientation. Initially the focal length was 
set at 45 nw. The camera offset parameters T and were obtained 
from direct measurements. These measurements were partly done 
using PUMA robot. A pointer of known dimension was attached to the 
end effector and this pointer was positioned at the appropriate 
locations. The VAL system then gives us the distances. It was 
found that (see (4.1)> 



-567.97 


X X -662.54 Y = 

o o 

r^= 61.821 r^ = -14.41 


Z^= 11.22 

r^ = 66.85<all mm > 


The angles e and a were measured using protractor and were 
found to be 


Pan (©> = 90*^ Tilt<a) =115*^ 

Once the transformation parameters were measured, the next 
step IS to verify these parameters. For this, a known world point 
< I .e. X, Y and Z are known) was imaged to obtain the image point 
<x.y) and <5.7 to 5.9) were used to verify the results. The 
following was observed : 


X ^=672.54 


mm 


m«a«ur«^d 


=-555.54 


mm 


Z 




-548.91 mm 


Image coordinates are x = 156 y = 215 (pixel units) 
scale in x-direction : 1 pixel = 0.0176 mm 

scale in y-direction : 1 pixel = 0.0115 mm 

From <5.7 to 5.9) X =560.65 mm Y . . = -460.80 mm 

computed computed 

The large errors in computed X and Y is unacceptable and 
another approach is desired to determine the transformation 
parameters accurately. It was noticed that <5.7. to 5.9) were 

sensitive to errors in 0 and a than the same percentage errors in 
the offset parameters. Therefore, these angles © and « were 
calculated by trial and error using measured i^^.r.Wand the image 
point <x,y). This process was repeated for seven known points and 
the results are presented in Table 4.1 <a). The mean of these 

angles was taken as the calibrated values for 0 and ot. and are 
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r^= 61.821 

Y^r-567.97 rj|»-U.41 

^o” n .22 r^= 66.85 


m 

Im«g« Coo rde 

H« i ght 

Direct Measurements 
Using PUMA-560 

Pan 

Tilt 

Focal 




Z<mm) 

(Final) 

1 ength 

■■ 

M' 

y 

X 

Y 

o 

9 

X (mm) 

1 „ 

141 

2S9 

-578 . 54 

715.81 

-528.47 

87,525 

112.875 

45.0 

2 ,, 

mm 

109 

-548.55 

757.22 

-505.78 

87.29 

112.845 

70.0 


^9 

140 

-548.55 

715.55 

“521.15 

87.42 

112.869 

70.0 

4 . 

13$ 

224 

-548.91 

672.54 

“555.54 

87.42 

112.8 

70.0 

5. 

137 

>48 

-578.22 

672.58 

“555.01 

87.42 

112.826 

70.0 

S . 

Z37 

>81 

“578.22 

660.91 

“562.06 

87.68 

112.698 

70.0 

7 . 

Z37 

245 

“548.81 

660.91 

-562.06 

87.68 

112.78 

70.0 






Mean Values 

1 

87.4907 

112.815 



TABLE 4# 1 Ca) Calibration 


Fo^C'.iil Ltngth X*40 mm 


SI m 

Iftmge 

Coords 

A1 t i tude 

2 

^ , • 
calc 

IH 

B 

Y 

d t r ec 1 

n 

y 

1 . 

165 

290 

“508.0 

556.17 

545.97 

“514.42 

-522.58 

2. 

245 

229 

If 

589.84 

585.58 

-557.85 

-555.55 

5. 

517 

267 

w 

552.97 

554.47 

-595.88 

-591.55 

4. 

246 

525 

If 

504.45 

515.75 

“555.82 

“561.54 


TABLE 4*1 Cb> Verification 
































e «= 87.4907 


and a = IIZ. 81507 

A further check was made to test the correctness of these 
parameters and the results are presented in Table 4.1 <b>. It can 
be seen that the results are satisfactory. 

In our work instead of rotating the camera to the new 
location the object is rotated to give the desired view. All the 
calculations to determine the location, orientation etc. are done 
with reference to the coordinates in the latest view. 

Example of a cxibo : 

The digitized image of the cube stored in the fTcans gi'ctbb&ir 
is shown in Fig 4.2. It is a raw image of the object without any 
pre processing. This is the input for the Low-vision module. 

First stage in the Low-vision module is odgo d»toction. We 
are using Sob&l* s operator from the I TEX library. The suitability 
of this edge detected image for further processing is limited by 
the illumination conditions and the scaling factor, i.e. sobel 
coefficient chosen (scaling factor would scale the output by 
dividing the pixel value with this coefficient). 

Effect of lighting is illustrated by Fig 4.5 which is the 
image of a badly lighted object. Fig 4.5 (a) shows the detected 
edges with a coefficient 45 in sobel operator. One can see clearly 
that some of the edges are missed out. This can be improved by 
better lighting conditions. By reducing the coefficient used for 
Sobel (25) we may be able to get the edge information from the 
same fig (4.5) but as can be seen in Fig 4.5 <b> there is a lot of 
noise introduced- This noise would hinder the further processing. 


Fig 4.2 is a carefully lighted image and we even accentuated 
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Fig 4.5 An example of badly Illuminated 
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some of its edges with a black marker to hei 
line diagram. This image is used for all the 
descr i bed . 


P us obtai 
Subsequent 


n a good 
operat i ons 


Let us see the effect of Sobel’s coeffi^i„„+ 

on the edge 

information. Fig 4.4<a) shows detected edooc i.i+K e 

'’'jges with a Sobel 

coefficient 25. This introduces lot of noise mab-inn c 4 . 1 . 

' making the further 

operations difficult. Fig 4.4(b> shows the 

uetected edges with 

sobel coefficient 45. One can see that all the edge information is 
present without any undesired noise. Fig 4.4<c) shows the detected 
edges with sobel 65. One can see that important edge information 
is lost. 

A Sobel coefficient between 35 and 45 was found to work best. 


After the edge detection, the edges are thinned to single 
pixel. Details of the thinning algorithm were presented in chapter 
2 . Fig 4.5<a> shows a thinned image of Fig 4.4(a>. 


Due to thinning some distortions are introduced especially 
near the corners as evident from Fig 4.5<b> which is the thinned 
image of Fig 4.4<b>. This problem can be handled better if we 
adopt different processing criteria at blobs, i.e. at all points 
at which lines either intersect or meet. The thinning strategy we 
followed fails when a line is perfectly horizontal. 

Some of these distortions (especially at the corners) are 
eliminated by the line finder to a certain extent as it finds the 
new points of intersections for lines meeting at a point. 

The thinned image is broken up into straight line 
segments. First procedure SEGMENT, divides the image into 
different contours. Procedure SEPARATE acts on these contours and 
breaks them up into valid line segments. At the end of these 
operations the thinned image transforms into a disjoint line 








Fig 4.5 (a) Thinning performed on Fig 4.4 (a) 



I 


Fig 4.5 (b) Thinning performed on Fig 4.4 <b> 
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diagram as shown in Fig 4.6<a>. 

Since the image represents a physical polyhedron. it is 
essential that the line diagram be continuous. The Hi-vision 
module can interpret only doubly connoctod edgo lists. To achieve 
this, all lines are extended till they intersect another line. If 
more than two lines are found to intersect within a specified 
neighborhood then a common point is assigned, at which all these 
lines are supposed to converge. Labeling, as explained in chapter 
2 is useful in solving the correspondences between the lines. Fig 
4.6 <b> shows a connected and labeled line diagram. 

Hi -vision iKxiule t 

Fig 4,7(a> &<b) show the line diagram formed for a cube 
from 2-different views and the corresponding DCELs. Thus the edge 
11 (labeled as til as it is a top edge) has a slope of 27.17 and 
the vertices have coordinates (243 229) and (317 267) in image 
plane . 


Initially ail the valid visual planes are formed and also the 
lines are grouped according to their parallelism. These planes are 
searched for the existence of grippable planes. When a view is 
inadequate another view is asked for and similar procedures are 
followed and the information base is updated. 
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Fis 4.6 (a) Disjoint Line 
Coutput of Fig 4.5<b> 


diagram 
was used 
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Fig 4.6 tb) Joined and Labeled Line Diagram 
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Fig 4.7 shows two views of the cube along with the associated 
DCELs . The execution summary for this example is given below. 


The valid planes are 

<(L7 L6 TL2 L5) (L3 L9 L8 TL5> <TL4 TL5 TLZ TLD) 

The Group of parallel lines is 

<<L9 TLl TL5) <L7 TLZ TLA) (L6 L8 L?>> 

"Looking for the grippable planes ... pi. wait" 

Another view is required 
Now analyzing the SECOND view 

The valid planes are 

<<L12 L8 TLA LIO) <L10 Lll L6 TLl) <TL2 TLl TLA TL9)) 

The Group of parallel lines is 

<<L11 TL5 TLl) <L12 TLA TLZ) (L8 L6 LIO)) 

The Combined plines is 

<<L9 Lll TL9 TLl) (L7 L12 TLA TL2> <L5 L8 L6 LIO)) 

The Combined vplanes is 

(<TL2 TLl TLA TL5) (LIO Lll L6 TLl) (LI 2 L8 TLA LIO) (L7 L6 TLZ L5) 
(L? L9 L8 TL9> (TLA TL> TLZ TL1>> 

"Looking for the grippable planes ... pi. wait” 

The grippable planes are 

<<L10 Lll L6 TLl) (L5 L9 L8 TL5)> 

(GO TO THE LOCATION 590 . 5A1 7292501 062 -554.3998655055534 -508.16 
152.0386355772563 90.0 0.0) 
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(a> First View <Iistl> 


(b) Second View (list2> 



Fig 4.7 Joined and Labeled Line Diagram of a CUBE 


(setq listl '<( til 27.170 ( 24J 229 >< 317 267 ) )( tl2 -39.230 ( 317 267 X 246 325 )| 

< 13 88.080 ( 246 325 )< 251 477 ) >< tl4 -38.012 < 243 229 )( 165 290 > | 

< tl5 23.360 ( 165 290 >< 246 325 ) )( 16 85.791 < 317 267 X 327 404 ) )! 

< 17 -43.829 < 327 404 X 251 477 ) X 18 86.785 ( 165 290 X 173 434 > )| 

< 19 28.856 < 173 434 X 251 477 ) >)) i 

(setg listZ *<< tl5 28.746 ( 236 223 X 318 268 ) X tl4 -39.274 ( 318 268 X 241 331 )| 

( 110 87.579 ( 241 531 X 247 475 ) X tl2 -42.340 < 236 223 X 168 285 ) 

< til 32.204 ( 168 285 X 241 331 > X 18 85.965 < 318 268 X 528 411 ) > 

< 16 87.611 ( 168 285 X 174 431 > X 111 31.067 ( 174 431 )t 247 475 > ) 

(112 -38.298 ( 328 411 X 247 475 ) ))) ' 

(setq Z -508.16) 


(c) Associated OCELs 



Fig 4.8 shows two views of the HEXAGONAL PRISM along with 
the associated DCELs. The execution summary for this example 
IS given below. 

The valid planes are 

<<L8 TL6 L7 LIO) (L8 TL5 L9 Lll) <TL5 TL4 TL5 TL6 TLl TL2)> 

The Group of parallel lines is 

<(L11 TL2 TL5) (TLl TL4> (LIO TL6 TL5> (L7 L9 L8)) 

"Looking for the grippable planes ... pi. wait" 

Another view is required 
Now analyzing the SECOND view 
The valid planes are 

((L12 L19 L15 TL5> (L12 L17 L9 L9A TL4) (TL6 TL5 TL4 TL5 TL2 TL1)> 
The Group of parallel lines is 

((L17 TLl TL4> (TL2 TL5) (L15 TL5 TL6> (L13 L9 L12>> 

The Combined plines is 

((L17 TLl TL4) (Lll TL2 TL5) (LIO L15 TL5 TL6> (L7 L8 L13 L9 L12)) 
The Combined vpianes is 

((TL6 TL5 TL4 TL3 TL2 TLl) (L12 L17 L9 L9A TL4) (L12 L15 LI? TL?) 
(L8 TL6 L7 LIO) (L8 TL5 L9 Lll) (TL? TL4 TL5 TL6 TLl TL2)) 

"Looking for the grippable planes ... pi. wait" 

The grippable planes are 

((L8 TL6 L7 LIO) (L12 L15 LI? TL?)) 


(GO TO THE LOCATION 968.9401279499559 -572. 9?57?8495857 -494.91 
50.68248687905665 90.0 0.0) 
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C «, > First V I «w < I i s 1 1 ) 


(b) Second View (iist2) 




Fig 4.8 Joined and Labeled Line Diagram of a 
HEXAGOMAL PRISM 


(»ctq Ilstl *<C tl2 -28.465 ( 227 75 )< 168 107 > >< til -86.952 < 168 107 )( 166 145 ) ) 

C 17 87.554 ( 166 145 >< 178 450 > )< 110 52.579 ( 178 450 )< 259 469 ) > 

< tl5 20.764 < 227 75 >< 285 97 ) >< tl4 81.641 < 285 97 X 291 158 ) > 

< tI5 -28.289 < 291 158 )< 226 175 ) X 18 87,450 ( 226 175 X 259 469 > ) 

< 19 89,567 ( 291 08 X 294 426 ) X 111 -58.004 < 294 426 X 259 469 > ) 

< tl6 25.007 < 166 145 X 226 175 ) )>> 

<setq Ii»t2 X< til 27.515 < 275 54 X 555 85 ) X tl2 88.865 C 555 85 X 554 157 > > 

< ti5 -24,955 ( 554 157 X 276 164 > X 112 88.527 ( 276 164 >< 285 445 > > 

C tiS -54.567 < 275 54 X 216 95 ) X tl5 88.416 ( 216 95 >< 217 150 > ) 

< 19a 59.012 < 217 150 X 226 145 > X 19 -89.755 < 226 145 >( 225 419 ) > 

< 117 22.470 < 225 419 X 285 445 > >< 115 88.527 < 554 157 X 541 416 > > 

< tl4 29.942 < 217 150 X 276 164 ) X 115 -24.955 < 541 416 X 285 445 > >) > 

<satq' 1 -454.91>;\ 


<c) Associated OCELs 
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rig 4.9 shows two views of the triangular prism in 
x.nfrtppabie con/itf^atton along with the associated DCELs . The 
execution surmiary for this example is given below. So. the system 
keeps asking for another view and after some views the analysis 
can be aborted. We are aborting it for four views. 


The valid planes are 

(<TL> TL2 TLl) <TL2 L4 L6 L5>> 

The Group of parallel lines is 
(<L? L4> <L6 TL2>) 

"Looking for the grippable planes ... pi. wait" 

Another view is required 
Now analyzing the SECOND view 

The valid planes are 

C(TL> TLl TL2> <L5 L8 L7 TLl) (L4 L9 L7 TL5)) 

The Group of parallel lines is 
<<TL? L9> (L5 L7 L4> <TL1 L8)) 

The Combined plines is 

<<TL? L9) <L5 L7 L4> <TL1 L8) <L6 TL2)> 

The Combined vplanes is 

((L4 L9 L7 TL5) <L5 L8 L7 TLl) (TL5 TLl TL2) <TL? TL2 TLl) 

<TL2 L4 L6 L5) ) 

"Looking for the grippable planes ... pi. wait" 

Another view is required 

THAT ’S HOW THE SYSTEM ASKS FDR ANOTHER VIEW SINCE IT CAN NOT ARRIVE 
AT A DECISION WITH THE AVAILABLE INFORMATION 
We say that the object is UNGRIPPABLE if 4 -VIEWS are insufficient to 
arrrive at a decision 
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(a) F irst View ( 1 istl ) 


ft>) Second View Clist2> 



Fig 4,9 Joined and Labeled Line Diagram of a 

TRIAHGULAR PRISM in Un^rippablB con/tl^wation 


tsttq list! M< til 29.16f < IIJ >< 290 299 ) )< tl2 -38.838 ( 290 299 >( 218 337 >' > 

C 14 87.271 ( 218 337 )( 222 442 ) >( tl3 87.131 ( 213 236 )( 218 337 > ) 

< 13 87.206 C 290 299 )( 294 382 ) )( 16 -39.790 ( 294 382 )( 222 442 ) >)) 

:setq listZ M< ti2 -21.397 < 324 242 >< 223 282 ) X 13 89.248 < 223 282 )< 224 362 > ) 

C 18 33.339 < 224 362 )( 261 416 > )( tl3 -33.133 ( 324 242 X 260 334 > ) 

C 17 89.263 i 260 334 X 261 416 > X 14 90.000 C 324 242 )< 324 323 ) ) 

C 19 -33.283 < 324 323 X 261 416 ) X til 34.543 ( 223 282 )< 260 334 ) )>) 

:setq 2. -327.88) 


<c> Associated OCELs 
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Fja 4.10 shows two views of the triangular prism in gr i ppcib I » 
onf iguraiion along with the associated DCELs . The execution 
sumrrtary for this example is given below. But since there do not 
exist two lines that are membe rs of the top plane the location and 
orientation for PUMA can not be determined. But two parallel 
planes are determined. 


The valid planes are 

<(L6 L5 L2) <TL1 L2 L4 L3>> 

The Group of parallel lines is 
<<L3 L2) <L4 TL2>> 

"Looking for the grippable planes pi wait" 

Another view is required 
Now analyzing the SECOND view 

The valid planes are 
<<L9 L5 L8> <TL1 L9 L7 L5>) 

The Group of parallel lines is 
<<L5 L9) <L7 TL1>) 

The Combined plines is 

L9> (L4 L7 TL1> <L? L2)> 

The Combined vplanes is 

<<TL1 L9 L7 L5) <L9 L5 L8> <L6 L5 L2> (TLl L2 L4 L5>) 

"Looking for the grippable planes ... pi. wait" 

The grippable planes are 
<<L9 L5 L8) (L6 L5 L2)> 

The lines (t> of interest don't exist in the latest case (2nd) 
i.e. There are a set of parallel planes but the LOCATION and 
ORIENTATION can not be determined because the planes do not have 
top lines < label starting with t ) in them. 




(setq listl •<( til 51.172 ( 169 225 )( 207 2A8 ) X 12 -69.566 ( 207 248 >( 160 575 ) ) 

< 15 -66.775 ( 169 225 >< 115 551 > X U 26.045 C 115 551 X 160 575 > ) 

< 15 47.796 < 207 248 X 265 512 ) X 16 -50.142 ( 265 512 )< 160 575 > ))) 

(setq ii8t2 ’<< til 27.159 ( 182 199 X 221 219 > X 15 47.796 < 221 219 )( 279 285 ) > 

( 18 -51.029 ( 279 285 X 171 548 ) X 15 -69.268 ( 182 199 X 154 526 > > 

( 17 50.725 < 154 526 X 171 548 > X 19 -68.786 ( 221 219 >< 171 548 > > ) ) 

(setq Z -529.41 > 


(c) Associated DCELs 



Chapter 5 Conclusions and Suooestions for Future work 


For the case of viewing single polyhedron the results have 
been encouraging. There were certain problems with the formation 
of the line diagram. But ascertaining the grippability and the 
actual gripping of the object has been successfully done. There 
are inaccuracies in determining the position and orientation but 
we are sure this aspect can be improved. (suggestions are given 
later in the chapter) 

The general approach in this thesis has been to get a 
solution to a simplified problem with tools tailor made to solve 
it. We consider only polyhedral objects and have only one object 
in the scene. We thus avoid solving the very much harder general 
vision problem. We do not think these methods can be generalized 
to solve the gripping problem for non-polyhedral objects. 

Even so the current approach has some limitations. These can 
be summarized under three headings : 

a) Low-vision : Largely image processing problems 

b) Hi-vision ; The algorithm which finds the valid visual 
planes may be improved. 

c) Grasp and Pre-grasp : Related to accurately determining 
the coordinates of the grasp planes and positioning and orienting 
the gripper correctly. 

Starting from the Low-vision module the, important step after 
getting a good edge detected image is to get the thinned image. As 
already mentioned in chapter 4 our approach distorts the image at 
corners and fails for horizontal lines. One mighty try using a 
di fferent strategy for blobs and the cases of horizontal lines. 
But multi-directional thinning and the logical AND OR operations 
between these images would be too expensive. 

The line finder will become faster if all the corners are 
determined directly rather than doing the seg mentation first and 
then breaking it into lines. Thus one can avoid the rescanning of 
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the entire ?12 x 512 


array involved 


in the segmenting process. 


A ftvaior lacuna is that we do not yet have a good theory 
of how to compute a new viewing location based on the data 
obtained in the previous views. In the experiments we carried out 
we were able to get answers by looking at the two views. The 
second view was arbitrarily chosen to be at 180** rotation of the 
camera about the vertical about the object. In general this is 
clearly insufficient and requires a properly articulated theory. 


There are large inaccuracies involved in determining the 
location in WCS from the image plane coordinates. This can be 
improved by improving the calibration. By improving the camera 
mount such that the camera offsets and the angles of pan and tilt 
can be read directly. As of now the position of the camera and the 
object should remain with in the robot's work envelope because the 
camera offsets etc. are measured using the robot itself. This 
resulted in the object being with in the minimum focus distance of 
the camera. This is one of the causes for large errors. But if the 
above mentioned mount can be made then proper focusing can be 
ensured. 


The d^pth or range data is not calculated in the current 
work. This is being fed to the module as explained in Chapter 3. 
The correspondence problem is being solved interactively by giving 
the lines absolute labels. This is a difficult problem and will 
require multiple views taken at small increments. Or this can also 
be done through a model based approach. 

The attached zoom lens (SONY) gives a variable focal length 
from 17.5 to 105 mm. It currently has no facility for auto-iris, 
fnotorized zoom auto-focus etc. which could be of great use if one 
has to move the camera to a different location for the second 


view. 
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APPENDIX A 


Modefl 


PULNiX TM-560 '6^3) 'ea4h^e^ 


SPECIFIC ATI WS 


Pixel H X V> 500 X 582 

Opt i ca 1 

Black H: 2(F> + 50<R> 

V: 14<F) 


Image SizeCmm) 
Dynamic Range 
Other 


8.8(H>x6.6<V) 

67 dB at 25°C 

Low noi se .blooming/smear ing suppression 


Scamnding 


625 lines (H) 
50 Hz <V) 2:1 
Interlace 


Clock Frequency 
Pixel Scan 
F requency 
Hor izontal 
Frequency 
Vertical 
Frequency 

Syrwc 

External Sync 

Hd 

Vd 

TTL level 
C~MOS level 


28.575 MHz 

9.458 MHz 

15.725 KHz 

50.00 Hz 

Auto Switch 
Negative Going pulse 
15.625 KHz (+5«) 
50.00 Hz <± 55K) 

(+5V to OV) or 
(OV to -5V) 
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Input impedance 4.7 K Ohm 

Interlace or Non-interlace 


TV Resolution 

Hor i zontal 


Ratio 

Sensitivity 

Miniinum 
i i I umi nat i on 

Video Output 

AGC 

Gamma 

Power 

Requi reiaants 

Lens Mount 

Auto~lris 


570 1 i nes 
(450 H*) 

420 1 i nes 

250 lines (non- interface ) 

50 dB type. {56dB with AGC off) 

50 lux (AGC off; F = 1.4) 

2 lux (AGC on; F = 1 ,4) 

0.6 lux type. (Gain Max; F = 1.4) 

l.OV p-p composite Video; Sync 
Negative (RS-170) 

12dB (On); OdB (Off) Internal Jumper 

Gamma = 0.45 (Standard) Pot 
adjustable to = 1 

12V DC ± 10%; Current 
Consumption; 500 mA 

Mini-Bayonet <C-mount with C-mount 
adaptor 

6-pin connector for PULNIX mini 
auto- iris lenses or standard EE 
lenses 



Temperature 

Range 

-10°C to + 50°C 

Camera Mount 

Direct ±y*-ZO mount hole on bottom 

of camera 

Si ze C mnD 

( i nches > 

42 <W> X 52 (H> X 119.5 <L> 

1 5/B (W) X 1 1/4 (H) X 4 a/4 <L> 

Weight 

200 g <7 oz) 

Vibration 

Shock 

7 G (11 Hz to 200 Hz) 

60 G shock 



APPENDIX B 


Hud&l PCVISION plus PicATie 

SPECI FI CATIONS 

Vi dec I nput 

, RS-170/530 or 50Hz CCIR (European) . NTSC luminance only 

. Two software-selectable video inputs 

. DC restoration corrects input signal drift 

. Eight 256 x 8-bit Input Look-Up Table allow pre-processing 

. Monotonic flash A/D Converter digitizes video at 10 MHz for 

512 X 480 <512 x 512) image, or 12.5 MHz for 640 x 480 image 
(square pixels) 

, Anti-aliasing filter provides - 5dB attenuation at 4.2 MHz 

and -12dB attenuation at 8 MHz 

. S i gnai -to-no i se ratio greater than 40 dB 

« Programnnab 1 e gain .67 to 1.55, and offset voltage -O.IV to + 

IV in 100 increments each 

. Programmed gain and offset voltage values are stored in a 

nonvolatile memory 

Frame Memory 

Frame memory size - 1024 x 512 pixels with 8 bits per pixel 


allows storage of two 512 innages or one 640 x 480 image 

. Memory-mapped access in 64K blocks 

. Simultaneous access of of up to eight pixels 

. Transparent computer access to frame memory - does not 
disturb display 

. Hardware zoom of 2:1 

. Hardware pan on eight pixel boundaries 

* Hardware scroll on two line boundaries 

. Bit planes protected independently from CPU access and video 
acqu i s i t i on 

Video Output 

. RS-170 or CCIR RGB video outputs with sync available on green 

channel 

. Output bandwidth: 10 MHz. minimum 

. Output voltage: IV p-p into 75 O load 

. Eight 256 x 8-bit Look-Up Table per output channel 

. Three 8-bit Video Frequency Digital-to-Anaiog Converters 

Timing and Synchronization 

. Software-selectable system clock source: crystal or 



« Two stage PLL provides stable horizontal lock to noisy 

sources, such as VCRs 

. Automatically switches to crysta 1 -generated timing if 

external sync is lost 

* Composite sync or separate horizontal and vertical sync 

outputs <TTL) to drive display and inputs 

. Pixel jitter typically less than 20 ns 

. Special sync bus for synchronizing multiple PCVISION plus 

modules for multiple-board configurations 

. Host access time; register access inserts no wait states, 

memory access inserts 0 to 120 ns of wait states for 512 

pixel per line image and 0 to 500 ns for 640 pixel per line 

image 

Host. Computer Interface 

. IBM Personal Computer AT, XT, and PC, and all 100% compa- 

tibles 

. 8-bit data bus, 24-bit address bus allows mapping into 

extended address space 

. 16 I/O mapped control registers mappable to any 16-byte 

boundary 


Power Requirements 



13 watts nominal 


+5V a 2.20A 

+12V 0.1 40A 

-12V a 0.030A 


Envi r onment al 

. Temperature-0-50 degrees Celsius 


Humidity - 10-90% non-condensing 



